Migrating HL compile and export to infer APIs #214
+102
−166
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Please rebase |
Change-Id: If27fbc1636ed1fe9b475d07cef7c83ed7dc46ca8 Signed-off-by: Asmita Goswami <[email protected]>
asmigosw
force-pushed
the
hl_compile_api_infer
branch
from
acf8ca3 to
12da558
Compare
QEfficient/cloud/export.py
Outdated
| ) # type: ignore | ||
| logger.info(f"Generated onnx_path: {onnx_model_path}, onnx_dir_path: {onnx_dir_path}") | ||
| logger.info(f"Exporting Pytorch {model_name} model to ONNX...") | ||
| qeff_model = QEFFAutoModelForCausalLM.from_pretrained(model_name, cache_dir) |
There was a problem hiding this comment.
We should not restrict CLI apis to use only AutoModelForCausalLM. It should be generic as we support new auto classes.
Change-Id: If27fbc1636ed1fe9b475d07cef7c83ed7dc46ca8 Signed-off-by: Asmita Goswami <[email protected]>
…transformers into hl_compile_api_infer
Signed-off-by: Asmita Goswami <[email protected]>
ochougul
requested changes
Jan 29, 2025
| hf_token: Optional[str] = None, | ||
| local_model_dir: Optional[str] = None, |
There was a problem hiding this comment.
Why are you removing this?
| @@ -92,7 +76,6 @@ def main( | |||
| model_name=model_name, | |||
| cache_dir=cache_dir, | |||
| hf_token=hf_token, | |||
| local_model_dir=local_model_dir, | |||
QEfficient/cloud/infer.py
Outdated
Comment on lines
107
to
115
| config = AutoConfig.from_pretrained(model_name) | ||
| architecture = config.architectures[0] if config.architectures else None | ||
|
|
||
| model_class = architecture_mapping.get(architecture) | ||
| if not model_class: | ||
| logger.error(f"Model class for model name {model_name} not found in mapping") | ||
| return | ||
|
|
||
| qeff_model = model_class.from_pretrained(model_name) |
There was a problem hiding this comment.
why? directly useQEFFAutoModelForCausalLM.from_pretrained
There was a problem hiding this comment.
Instead of writing own dictionary please make use of
MODEL_FOR_IMAGE_TEXT_TO_TEXT_MAPPING_NAMES , MODEL_FOR_CAUSAL_LM_MAPPING_NAMES from transformers.
from transformers.models.auto.modeling_auto import MODEL_FOR_IMAGE_TEXT_TO_TEXT_MAPPING_NAMES, MODEL_FOR_CAUSAL_LM_MAPPING_NAMES
## Now check if the architecture is present in either of the values of these two dictionaries and call our corresponding auto class based on that.
QEfficient/cloud/infer.py
Outdated
Comment on lines
19
to
32
| # Map model's architecture to class | ||
| architecture_mapping = { | ||
| "LlamaForCausalLM": QEFFAutoModelForCausalLM, | ||
| "GPT2LMHeadModel": QEFFAutoModelForCausalLM, | ||
| "MistralForCausalLM": QEFFAutoModelForCausalLM, | ||
| "FalconForCausalLM": QEFFAutoModelForCausalLM, | ||
| "GPTJForCausalLM": QEFFAutoModelForCausalLM, | ||
| "GemmaForCausalLM": QEFFAutoModelForCausalLM, | ||
| "Gemma2ForCausalLM": QEFFAutoModelForCausalLM, | ||
| "Phi3ForCausalLM": QEFFAutoModelForCausalLM, | ||
| "Qwen2ForCausalLM": QEFFAutoModelForCausalLM, | ||
| "GPTBigCodeForCausalLM": QEFFAutoModelForCausalLM, | ||
| } | ||
|
|
Signed-off-by: Asmita Goswami <[email protected]>
ochougul
requested changes
Jan 29, 2025
QEfficient/cloud/export.py
Outdated
|
|
||
| from QEfficient.exporter.export_hf_to_cloud_ai_100 import qualcomm_efficient_converter | ||
| from QEfficient.utils import check_and_assign_cache_dir, onnx_exists | ||
| from QEfficient.transformers.models.modeling_auto import QEFFAutoModelForCausalLM |
There was a problem hiding this comment.
from QEfficient import QEFFAutoModelForCausalLM it's present in __init__
QEfficient/cloud/export.py
Outdated
| full_batch_size=full_batch_size, | ||
| ) # type: ignore | ||
| logger.info(f"Generated onnx_path: {onnx_model_path}, onnx_dir_path: {onnx_dir_path}") | ||
| logger.error(f"Model class for model name {model_name} not found in mapping") |
There was a problem hiding this comment.
`raise NotImplementedError(f"Unknown architecture={architecture}, either use specific auto model class for loading the model or raise an issue for support!")
There was a problem hiding this comment.
We should fail here which will force the script to exit
QEfficient/cloud/export.py
Outdated
| ) # type: ignore | ||
| logger.info(f"Generated onnx_path: {onnx_model_path}, onnx_dir_path: {onnx_dir_path}") | ||
| logger.error(f"Model class for model name {model_name} not found in mapping") | ||
| return |
QEfficient/cloud/infer.py
Outdated
| enable_qnn=enable_qnn, | ||
| qnn_config=qnn_config, | ||
| ) | ||
| logger.error(f"Model class for model name {model_name} not found in mapping") |
QEfficient/cloud/infer.py
Outdated
Comment on lines
94
to
108
| config = AutoConfig.from_pretrained(model_name) | ||
| architecture = config.architectures[0] if config.architectures else None | ||
|
|
||
| if architecture in MODEL_FOR_CAUSAL_LM_MAPPING_NAMES.values(): | ||
| model_class = QEFFAutoModelForCausalLM | ||
| else: | ||
| # Handle onnx model generation | ||
| onnx_model_path = get_onnx_model_path( | ||
| model_name, cache_dir, tokenizer, hf_token, local_model_dir, full_batch_size | ||
| ) # , base_dir_name) | ||
|
|
||
| ######### | ||
| # Compile | ||
| ######### | ||
| _ = QEfficient.compile( | ||
| onnx_path=onnx_model_path, | ||
| qpc_path=os.path.dirname( | ||
| qpc_dir_path | ||
| ), # We need to pass parent directory of qpc_dir_path, as the compile function handles the qpcs directory creation | ||
| num_cores=num_cores, | ||
| batch_size=batch_size, | ||
| prompt_len=prompt_len, | ||
| ctx_len=ctx_len, | ||
| mxfp6=mxfp6, | ||
| mxint8=mxint8, | ||
| aic_enable_depth_first=aic_enable_depth_first, | ||
| mos=mos, | ||
| device_group=device_group, | ||
| full_batch_size=full_batch_size, | ||
| allow_mxint8_mdp_io=allow_mxint8_mdp_io, | ||
| enable_qnn=enable_qnn, | ||
| qnn_config=qnn_config, | ||
| ) | ||
| logger.error(f"Model class for model name {model_name} not found in mapping") | ||
| return | ||
|
|
||
| qeff_model = model_class.from_pretrained( | ||
| pretrained_model_name_or_path=(local_model_dir if local_model_dir else model_name), | ||
| cache_dir=cache_dir, | ||
| hf_token=hf_token, | ||
| full_batch_size=full_batch_size, | ||
| ) |
There was a problem hiding this comment.
Since this code is a copy of the same in export method you can create a common method in utils file in cloud folder and use from there. you can call it.
load_qeff_model
Signed-off-by: Asmita Goswami <[email protected]>
Signed-off-by: Onkar Chougule <[email protected]>
Signed-off-by: Onkar Chougule <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Migrating HL compile API and export API to infer APIs